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ABSTRACT 

A Conference Role Semantic Differential was dev'^loped 
and field-tested as a brief, diagnostic measure of conference role 
performance of administrators. Principal components analysis of 52 
staff member •s evaluations of three institutional researchers yielded 
one task and three interpersonal skill factors which demonstrated the 
construct validity of the instrument. Cronbach»s Alpha yielded in 
internal consistency reliability coefficient of .86 for the 
instrument. Yeldman's "relate" procedure indicated substantial 
stability of factors over one year. After revisions suggested by 
these analyses, this semantic differential instrument should be 
helpful in administrator development, contract specification or 
renewal and salary determination at all levels of education. 
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Abstract 



A Semantic Differentia! Instrument 
for 

Evaluating Conference Role Performance of Educotional Administrators 



by 



Dr. Alfred R. Hecht, Director 
Office of Research and Evaluaiion 
Moraine Valley Community College 

P^los Hills, Illinois 60A65 



A Conference Role Semantic Differential was developed and field-tested 
as a brief, diagnostic measure of conference role performance of administrators. 

Principal components analysis of 52 staff member's evaluations of three 
institutional researchers yielded one task and three interpersonal skill fac- 
tors which demonstrated the construct validity of the instrument. 

Cronbach^s Alpha yielded an interr.al consistency reliability coefficient 
of ,86 for the instrument. Veldman's "relate" procedure indicated substantial 
stability of factors over one year. 

After revisions suggested by these analyses, this semantic differential 
instrument should be helpful in administrator development, contract specifica- 
tion or renewal and salary determination at all levels of education. 
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A Sciuiaiic DUferentiui InsLrunenc tor ' 
Evaluating Coni\-rence Rolv Per forr.ianec of EducM ional Administrators 

A. growing concern for administrator t^valujiion has accompanied tiu- educa- 
tional accountability movement. On th<' basis of a review of the ERIC litera- 
ture on evaluation, I'oiiakoff (1973) r^porti'd that "by 1968 a growing trend 
to evaluate school administrators was t'vid«.nt." 

Although writers such as 3arraclough (1973), McCleary (1973) and Melton 
('1970) cite the need for devt^lopmt^nt of a variety of measures of increasingly 
complex administrative roles at all Itrvels of education, according to 
Barraclough (1973) and Scriven (1974), procedures for evaJuating administrative 
performance are underdeveloped. 

In this pilot study, a Conference Role Seinan tic Differential (CRSD) was 
developed and field tested as a brit f uiap,nostii^ measure of this educational 
leadership role. 

Revievv^ of J^i t era ti|_r_i^ 

The need for valid and rt liable, uiagnostlt measures of administrator 
perforaaiive was cited by Castetter and iteisler (J071), Rosenberg (1971), and 
by Wochner and Lynch (1973). 

In contrast to the multitude of studie.s on technical characteristics of 
facuJtv performance measur<.'S, stuaies of adirunistrative evaluation lack such 
characteristics. Instead, they focus on i detR I f j cat i on of personnel evaluated, 
frequ^L-ncy of evaluation,' evaii'-.:on forms or Inst run.Mits , not i f i at i on of re- 
.'..ultr. and appeal p rocediirt's . Stemnucl:, for ..-.xarrpK', b.as rcporu-d four studies 
of administrcit i ve performance for the Educational Rc^.scarch Service. her J968 
study included the formalized evaluation proL-durts of 62 sciiooJ systeir.s. Her 
1970 report presented the "cl ien t-or u nl ed" evaLu<u ion procedures of 29 schooi 
systems. In 1971, Siennock^s report included t valuation form.s of 11 school 
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syst«:r,s. Iru^^*^ ivTriS ^^r. iniendod lu sUmulaic ih^ chinking of individuals 
Involved in dL-veloping proc^dires for cvalualint; adininis i raters . In her 1973 
repori, she summarizes tht. ci lent-cent ered— j-valuallon procedures of 469 school 
systen.^;. Thi.^ repcr. li. eludes IT administrator evaluation forms. In 1973, 
Napa College described its procedures for evaluating college adminis trat ors » 
However, none of these studies included data on liie technical characteristics 
of any of the ins trume^n t s presented. 

Because of its ability to measure complex concepts by means of a simple 
format, the semantic differential has been used widely in educational research. 
In studies of teacher effectiveness, the semantic differential has been shown 
to possess desirable technical characteristics* For example, Gulo (1972) com- 
pared trie results of four studies in which tlie semantic differential was used 
to evaluate the teaching .effectiveness of colh*ge professors* He concluded: 

• some teacher effectiveness factors are stable across time and 
populations 

• the proportion of variance accounted for by each factor varies 
across populations and from one factor analysis to another 

• the semantic differential seems to be an especially useful 
technique for quantifying emergent variables associated with 
student perceptions cf teaching effectiveness and effective 
professors . 

In addition to its use in faculty ev.Aluation, the .^.emantic differential has 
been used in administrator evaluation. 

Aliiiough no data on its techni(*al charae t .^r i s li c s are given, Stemnock^s, 
1970 report ineluded a semantic differential which has been used to (.valuate 
principals in Cheyenne, Wyoming. This instrum».nt, which consisted of 18 adjec- 
tive and Verb-phrase pairs, provides a global ass<'ssment of the principal's 
role , 



Proce d ureB 

PrompLed by the semantic di f f crorit ia3 ust d^for teacher evaluation of prin- 
cipals in Cheyenne, Wyoming (Steninock, 1970, p. 42), a Confere nc e Role Semantic 
Differential (CRSD) CF'gure 1) was designed to sample task and interpersonal 
functions of tducational leaders in individual conferences. Seven bi^polar 
adjective pairs were selected to represent each function. Four adjective- 
scales were drawn from the Cheyenne instrument. The remainder v/ere drawn or 
adapted from Osgood (1937, pp. 53-61) using his criteria for scale selection 
(pp. 78-80). 

Item order and adjt^ctive polarity were randomized to minimize response 
bias (Heise, 1970). A seven interval respons*- svale was used with each in- 
terval labelled with an adverb to define the sc<}le positions (Weils and Smith, 
1960). 



Insert Figure 1 about her*. 



In addition to the 14 adject ive-scalt s , tJie CRSD included two free-response 
items which asked evaluators to identify strengths and weaknesses in the con- 
feren'-^e roles performed by the person being fvaluattd. An unsigned draft CRSD 
was completed by 52 staff members. In the judgment of three inst i tutioaal 
research staff members each t valuator had sufficient ^.onference experience to 
qualify him for th*. task. Responses lr(;m all 52 individuals were pooled for 
factor analysis of construct validity (i5ashook and Foster, 1973). 

Oa the basis of the factor an<i lysis, tact(U' sci^res were calculated by 
summing across the weighted responsi -> for tin statements c<nTiprising -ach 
factor.. For each factor and for tne entire seal", internal consistency reli- 
ability was determmec according to CronbaLii^s Alpiui (Yonker, Blixt and Dinero, 
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1974). In addition, facior stability was determined by applying Veldman's 
(1967, pp. Ikl-lkk) "relate procedure" to the 1972 and 1973 CRSD factor 
loadings (Bashook and Foster, 1973). 

All statistical calculations were performed by an IBM 370 computer using 
programs TESTAT, FACTOR and RELATE (Vcldman, 1967, pp. 174-180, 222-236 and 
242-244). 

A revised CRSD (Figure 2) was administered one year after the .Irafl scale. 



Insert Figure 2 about here 



R esults 

As shown in Table 1, for the draft CRSD, principal components analysis 
with varimax rotation to simple structure yielded four factors--one task, 
two interpersonal and one task and interpersonal. 

Insert Table 1 about here 



In an effort to obtain clearer factors, CRSD was revised by omitting an 
adjective-scale which loaded almost equally on two factors and by adding 
three adjective-scales intended to help define three factors. As shown in 
Table 2 principal components analysis of varimax rotation to simple struc- 
ture for the revised CRSD yielded one task and three interpersonal factors 
which accounted for 70 per cent of vari.mce. The loadings suggested 

these factor descriptions: Problem-Solving, TacL in Personal Contacts, 
Accepting Others* Views and Persuasive. Tlu^se factors arc similar to several 
established by Brown (1967) in his study of job <nicjlysis by multidimensional 
seal ing. 

7 
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Insert Table 2 iibout horc 



Although these factors were clearer than those obtained with the draft 
CRSD, shifts in factor location for some sc<'*les indicate a need for further 
validation on a larger sample. 

Internal consistency reliability of CRSD factors ranged from .39 to .91 
with the factor comprised of the largest number of items having the largest 
coefficient. The revised CRSD instrument had an alpha of .86 as shown in 
the following table. 

Internal Consistency Reliability of CRSD 



-'number Coefficient 
Factor of Items Alpha 

Tact in Personal Contacts 7 .91 

Accepting Others* Views 4 .64 

Persuasive 2 .63 

Problem-Solving _2 / .39 

CRSD 15 .86 

N- = 52 



Factor stability from 1972 to 1973 was t^stablished by applying Veldman's 
(1967) ^'relate procedure" to maximize the fit butween 1972 and 1973 factor 
soluLions. Thf cosines of the angles betwtiun tht factors produced in the 
"relate procedure" are shown in the following table. These cosines can be 
intti-rpre ted in the samr way as correlation co^- f f i c i tnts betwc-en factors. 





Stabi b 


ty of CRSD 


Factors 




1972 Factors 




1973 


FacJ ors 


* 




1 


2 


3 




1 


^68 


-.221 


-.438 


. 149 


2 


.118 


-.299 


.625 


^712 


3 


.496 


.534 


^579 


-. 366 


/♦ 


-.053 


.760 


-.288 


.580 
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Factor i in the 1972 solution is substantially correlated with Factor 1 
in Che 1973 solution, and these factbrs are jomparatlvoly uncorrelated with 
the other three factors. 

Factors 2 and ^ i . these solutions are Interchangeable. 

/ 

Although the third factor in the 1972 solution is related to the third 
factor in the 1973 solution, the third factor of the 1972 solution is also 
related to the second factor and somewhat /to the first factor of the 1973 
solution. I 

These results indicate a substantial; stability of CRSD factors from 1972 
to 1973. • ' 

The adjective-scale vector correlat'ions shown in Table 3 indicate that 

thc: differences in "meaning" of tht? factors derived in the two analyses are 

/ 

attributable largely to the adject iverscales "Approachable-Unapproachable," 

/ 

"Accurate-Careless" and "Pleasant-Annoying" all of which changed factor 

/ 

locations from one solution to the (Jther, 

/ 



Insert Table 3 about here 



/ Summary 

This pilot study has described the initial development of a Conference 
Role Semantic Differential (CRiSD) as a measure of this role which is common 
ro administrators at all educational levels. 

The CRSD reported m this study shows substantial construct validity and 
a factor structure which accounts for 70 per cent of the common variance. 
In addition, the CRSD has an acceptable internal consistency reliability of 
.86 and the general stability of its factors over time has been demonstrated. 
However, the internal consistency reliability of the factors should be in- 
creased by adding related scales to permit diagnostic use of factor scores. 
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Scale subSLilulions or modifications should be made, also, to further increase 
the independence of the factors before this instrument is used widely. 

Other limitations of this instrument also reflect its developmental stage. 
These include: 

• Use of only three community college institutional researchers as 
the sample evaluated. 

• Use of only 52 community college staff members as evaluators. 

• Lack of norms for interpretation of relative performance. 

In spite of these present limitations, refined versions of this instrument 
should be helpful in staff development, contract renewal and salary determina- 
tion. Because of the universality of the specific educational role assessed 
by this instrument, it should be applicable to all levels of administrators 
at all levels of .cation. 
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FIGURE 1. Conference Role Semantic Differential (Draft) 



STAFF EV<VLUATION OF 



PART I. Directions: 



P"* -^cc an '^X" on each line below, at the point which 
represents your judgment of this staff member. 



unapproachable l 
annoying | 
approving I 



attentive (_ 



closed-minded |_ 



constructive ideas l 



impulsive l 
accurate (_ 
gets to point | 
superficial 
prompt |_ 
ignores problems i 



No 



Very Quite Slightly Op 



± JliliiL Quite Very 



J approachable 
J pleasant 
J faultfinding 
J inattentive 
J open-minded 

j worthless ideas 



J deliberate 



J careless 



I roundabout 



J analytical 



J slow 



J solves problems 



PART II. Directions: Answer in the spaces below. You may want to describe 

an incident to illustrate your comments. 



i. WHAT CONTRIBUTES MOST TO THIS STAFF MEMBER'S JOB EFFECTIVENESS? 



2. IN WHAT WAYS COULD THIS STAFF MEMBER INCREASE HIS JOB EFFECTIVENESS? 
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*^*It;UR£ 2, Conference Role r^taiantic Differential (Revised) 



STAFF EVALUATION OF 



I, Directlono: pi-^ c an "X^' on each line below, at the point which 
represents your judgraent of this staff meinber. 

No 

Very Quite Slightly Opinion Slightly Quite , Very 



accepting 
unapproachable 
annoying 
approving 
attentive 
closed-minded 
constructive ideas 
impulsive 
accurate 
gets to point 
superficial 
prompt 
ignores problems 
dictates 
clarifies 



rejecting 



J approachable 



J pleasant 



J faultfinding 
I inattentive 



J open-minded 



J worthless ideas 



J deliberate 



J careless 



J roundabout 
J analytical 



J slov; 



J solves problems 
J persuades 
1 confuses 



eART il. Directions: Answer in the spaces below. You may want to describe 

an incident to illustrate your comments. 

1. 'Au\T CONTRIBUTES MOST TO THIS STAFF ^^E^1BER'S JOB EFFECTIVENESS? 



2. IN WHAT WAYS COULD THIS STAFF MEMBER INCREASE HIS JOB EFFECTIVENESS? 
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TABLE 1. 1972 RotaLed Tacia Loadings fo 

Research 



, Factors 
I. Person-Task 

Deliberate — Impulsive 
Analytic^ i. — Superficial 
Approachable — Unapproachable 
Attentive — Inattentive 
Cooperative — Uncoop- rative 



II. Problem-Solving 

Accurate — Careless 

Solves Problems — Ignores Problems 

Constructive Ideas — Worthless Ideas 



III. Tact in Personal Contac*:s 

Gets to Point — Roundabout 
Pleasant — Annoying 
Prompt — Slow 

Cooperative — Uncoope rative 

IV* Accepting Others* Views 

Approving — Faultfinding 
Open-Hinded — Closed-Minded 

Per Cent of Total Variance 



a Leading decimals have been omitted 



■ Conference Roles of Institutional 
Personnel 



T a 


T T 
11. 


T T T 
III. 


iV . 


Loromunaj 


866 


- 31 


11 


j15 


ODD 


853 


310 


89 


29 


832 


7A1 


230 


435 


- 91 


780 


701 


161 


79 




0 


536 


244 


545 


- 28 


688 


31 


885 


-113 


"225 


847 


318 


865 


175 


72 


886 


289 


825 


O O "7 

Zo7 


T Q 
/O 


OJ J 


ijy 






— 


O J J 


- 23 


81 


714 


333 


629 


0 "7 o 


67 


o6't 




/ u ^ 


536 


Ikk 


545 , 


28 


688 


- 28 


37 


172 


923 


884 


119 


402 


27 


771 


771 


40 


16 ' 


13 


9 





from all factor loadings 
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TABLE 2. 19^73 Rotated FacL..r i.uadings lor Conference Roles of Institutional 

Research Personnel 



Factors 


I. ^ 


IT. 


III. 


IV. 


Communa 


Tact in Personal Cor.t * t^. 












Gets to Point — Roundabout 


908 


90 


8 


140 


852 


Clarifies — Confuses 


o c / 


136 


T O 

7 J 


280 


QO O 
OJO 


Attentive—Inattentive 


851 


16 


86 


129 


748 


Pleasant — Annoying 


841 


343 


201 


116 


878 


An al V t i c al — S upe r f i c i al 


778 


50 


237 


224 


714 


Prorap t — S low 


coo 

530 


OCT 

257 


o / c 

345 


3o6 


600 


Accurate — Careless 


504 


18 


427 


343 


555 


Accepting Others* Views 












Accepting — ReJ ectlng 


30 


727 


1 7 


19 


530 


Approachable — Unapproachab le 


438 


615 


94 


87 


585 


Approving — Faultfinding 


80 


584 


90 


51i 


619 


Open-Minded — Closed-Minded 


351 


562 


211 


530 


764 


Persuasive 












Persuades — Dictates 


36 


361 


830 


16 


821 


De Liberate — Impulsive 


443 


137 


740 


4 


762 


Prob leu -Solving 












Constructive Ideas—Worthless Ideas 


100 


116 


163 


798 


686 


Solves Problems — Ignores Problems 


291 


117 


238 


606 


, 522 


Per rent of Total Variance 


32 


13 


12 


13 




^ Leading decimals have been omitted 


from all 


f actor 


loadings 
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Table 3. Adjective Scaie-Vector Correlations and Factor Locations 



Factor Location 



Adjective Scale 


r 


1972 


1973 


Approacnable — Unapproachable 


.501 


1 


2 


Accurate — Careless 


.696 


2 


1 


Pleasant — Anncying 


.713 


3 


1 


Constructive Ideas — Worthless Ideas 


.781 


2 


4 


Gets to Point — Roundabout 


.793 


3 


1 


Deliberate — Impulsive 


.873 


1 


3 


Prompt — Slow 


.880 


1 


3 


App ro vin g — Paul t f inding 


.908 


4 


2 


Attentive — Inattentive 


.916 


1 


1 


Open-Minded — Close-Minded 


.922 


4 


2 


Analytical — Superficial 


.941 


1 


1 


Solves Problems — Ignores Problems 


.986 


2 


4 



ERIC 



17 



\ 



